AN ALMOST SURE INVARIANCE PRINCIPLE FOR ADDITIVE 
FUNCTIONALS OF MARKOV CHAINS 

F. RASSOUL-AGHA AND T. SEPPALAINEN 

Abstract. Wc prove an invariancc principle for a vector- valued additive func- 
tional of a Markov chain for almost every starting point with respect to an ergodic 
equilibrium distribution. The hypothesis is a moment bound on the resolvent. 



1. Introduction. 

This note extends a result of Maxwell and Woodroofe . Our notation and pre- 
sentation follow as closely as possible, and some results from there will be repeated 
without proofs. The work presented here was motivated by applications to random 
walk in random environment that are reported elsewhere. 

After completing this note we learned of the work of Derriennic and Lin on frac- 
tional coboundaries of Banach space contractions j^]- The estimates needed for the 
invariance principles we prove can be then obtained by applying the Derriennic and 
Lin machinery, and this way one can even improve the moment hypothesis to just 
having two moments (p = 2 below); see Thus, currently our note offers alternative 
probabilistic proofs of the results of |3] under the more restrictive moment hypothesis 



of @. 

Let {Xn)n>o be a stationary ergodic Markov chain defined on a probability space 
P), with values in a general measurable space {X,B). Let Q{x]dy) be its 
transition probability kernel and n the stationary marginal distribution of each X^. 
Write E for the expectation under P. denotes the probability measure obtained 
by conditioning on Xq = x, and is the corresponding expectation. For p > 1, we 
will denote by L^iii) the equivalence class of i3-measurable functions g with values 
in M.'^ for some d > 1 and such that 

yifp = J |fi'(a;)|%(c?x) < oo. 

Here, I ■ I denotes the £^-norm on M"'. 
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Now fix d and an M'^-valued function g e L^{7i) witli / g dr: = 0. Define SQ{g) = 
and 



We are concerned with central limit type results for Sn{g) and Sn{g)- This question 
has been investigated from many angles and under different assumptions; see and 
its references. A widely used method of Kipnis and Varadhan j3| works for reversible 
chains. Article adapted this approach to a non-reversible setting, and used growth 
bounds on the resolvent to obtain sufficient conditions for an invariance principle for 
Sn{g) under P, if p> 2. 

Derriennic and Lin used then their theory of fractional coboundaries [3] to push 
the result to an invariance principle for Sn{g) under P^, for vr-a.e. x, even when p = 2; 
see jsf. Using their method one can also show that the same almost-sure invariance 
principle holds for Sn{g)- We will show how to further the probabilistic technique of 
to yield both almost-sure invariance principles (for Sn{g) and Sn{g)) when p > 2. 
Invariance principles for additive functionals of Markov chains have many appli- 
cations. This note is a byproduct of the authors' recent work on random walks in a 
random environment 0, Q where this invariance principle proved useful. 

Let us now describe the structure of this note. In Section |21 we will present the 
setting of and prove an L'^ bound, with g > 2, on a certain martingale. In Section 
01 we will state and prove the main theorem of the note. The proof depends on a 
vector-valued version of a well-known invariance principle for martingales (Theorem 



Q is a contraction on for every p>l. For e > let /i^ be the solution of 



n 



Sn+i{,g) = ^g{Xk) and Snig) = Snig) - ExaiSn{g)), for n>0. 



3 of 0). 



2. A USEFUL MARTINGALE. 



For a function h G -^^^(Tr) and vr-a.e. x E X define 




(1 + e)he - Qhe = g. 



In other words. 



oo 



hs = Y,i^ + er'Q'^'g. 



k=l 



Note that G ^^(vr), if g e ^^(vr). On X'^ define the function 

H^{xo,Xi) = he{xi) - Qh^{xo). 
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For a given realization of {Xk)k>o, let 



n-1 



Mn{e) = J2He{Xk,Xk+i) and Rn{e) = Qh,{Xo) - Qh,{Xn) 

k=0 

SO that 

Sn{9) = Mn{e)+eSn{he)+Rn{e). 
Finally, let tti be the distribution of {Xq,Xi) under P; that is 

ni{dxo,dxi) = Q{xo; dxi)n{dxo) . 

Let us denote the L^-norm on LP(7ri) by ||H||p. The following theorem summarizes 
results of j^. 

Theorem MW. Assume that g G L'^i'ir) and that there exists an a E (0, 1/2) such 
that 



n-1 



a(n°). (2.1) 



9 

k=0 

Then 

(a) The limit H = lime^o+ exists in L^(7ri). Moreover, if one defines 

n-1 

Mn = H{Xk, Xk+l), 

then, for TT- almost every x, (M„)„>i is a P^-square integrable martingale, rel- 
ative to the filtration {Tn = o"(Xo, ■ ■ ■ , Xn)}n>o- 
(h) One has \\he\\^ = O'ie'"'), and if Rn = Sn{g) - Mn = Mn{e) - Mn + eSn{he) + 
Rn{s), then 

E{\Rn\^) = a(n2°). 

Proof. The existence of H follows from Proposition 1 of The statement about 
Mn follows from Theorem 1 therein. The bounds on \\hs\\2 and E[\Rnf) follow from 
Lemma 1 and Corollary 4 of respectively. □ 

If, moreover, one has an assumption on g, then one can say more. 

Theorem 1. Assume that there exists ana < 1/2 for which (j2.1|) is satisfied. Assume 
also that there exists a p > 2 such that g G L^^tt). Then there exists a q E (2,p) such 
that H G L'^{t^i) and (M„)„>i is an L'^ -martingale. 

Proof. First choose a positive g < (3 — 2a)p/ {1 — 2a + p). One can check that since 
2a < 1 and p > 2, we have q G (2,p). Using Holder's inequality, we have 

\\\TJ U ^ \\\ U Ulll^lllfT' U \\\^ 
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where a = p{q — 2)/{p — 2) < q and b = q — a. Next, observe that 

\\hsl<J2(^+er-\\9l=\\gle~\ 



n>l 

Thus, one has 



111^^6 - H,\\\l < 2- \\g\\; {e-' + d-'r\\\Hs - H,\\\l, 
and, by Lemma 2 of 

\\\Hs,-Hs,_X < C2'^.2-'''/\\\hs£+\\hs,_Xf\ 

where 6k = 2~^ . By part (ii) of Theorem MW, we know that Ij/i^Hg = 0"((5~°). 
Therefore, one has 

\\\Hs,- Hs,_^\\\l < C2'=("-^/2+a6)^ 

with maybe a different C than above. Now, by the choice of cl one can verify that 
a — b/2 + ab < 0, and then repeat the proof of Proposition 1 in jsf, with |||-|||2 replaced 

by Ill-Ill,. □ 

Remark 1. Note that uses ||-||-,^ for the L^-norm under tti, while we use ||H||2- 



3. The almost sure invariance principle. 

First some notation. We write for the transpose of a vector or matrix A. An 
element of M*^ is regarded as a c? x 1 matrix, or column vector. Define 

B„(t) = n-'/^S[rr4g) and B„(t) = n~'/%^t^{g), for t G [0, 1]. 

Here, [x] = max{A; G Z : A; < x}. Let D]gd([0, 1]) denote the space of right continuous 
functions on [0, 1] taking values in and having left limits. This space is endowed 
with the usual Skorohod topology 0]. Let A denote the Prohorov metric on the space 
of Borel probability measures on D^d{[0, 1]). 

For a given symmetric, non-negative definite d x d matrix F, a Brownian motion 
with diffusion matrix F is the M'^- valued process {W{t) : < t < 1} such that 
W{0) = 0, W has continuous paths, independent increments, and for s < t the d- 
vector W{t) — W{s) has Gaussian distribution with mean zero and covariance matrix 
{t—s)r. If the rank of F is m, one can produce such a process by finding a.dxm matrix 
A such that F = AA^, and by defining W{t) = AB{t) where B is an m-dimensional 
standard Brownian motion. 

Let $r denote the distribution of Brownian motion with diffusion matrix F on the 
space DiRd([0,l]). For x E X let respectively be the distribution of 

B„, respectively B„, on the Borel sets of D]gd([0,l]) under the measure Px] that is, 
conditioned on Xq = x. 

Here is our main theorem. 
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Theorem 2. Assume there are p > 2 and a < 1/2 for which g G I^^(vr) and 
E{\Rl\) = (7(n2"). Then 

lim A($j), \l/„(x)) = for n-a.e. x, 

n— »oo 

where D = E{MiMf) = J HH^dni. 

Remark 2. The above result improves Theorem 2 of which stated that 

hm / A($j,,^„(x))7r(da;) = 0. 

Remark 3. Due to Theorem MW, ()2.H) guarantees the bound on E{\Rn\'^) in The- 
orem |21 

Proof. The proof is essentially done in We explain below how to apply Borel- 
Cantelli's Lemma to strengthen their result to an almost sure statement. 
Let M*{t) = n-^'^M^nt]- We have 

sup |B„(t) - M*(t)| < n-'^'^ max\Rk\ . 

0<t<l k<n 

Therefore to conclude the proof we need to show two things: 

for vr-almost every x, under the probability measure the processes 

M* converge weakly to a Brownian motion with diffusion matrix D, (3.1) 

and 

n"^''^ max — > in P^j-probability, for vr-a.e. x. (3.2) 

k<n n— »oo 

Statement ()3.1|) follows from the martingale invariance principle stated as Theorem 
3 in 0]. The limits needed as hypotheses for that theorem follow from ergodicity and 
the square-integrability of H. We leave this check to the reader. 

To prove (j3.2p . let Uj = for a large enough integer r. Fix < 7 < 1, and let 

.j = \n^j~^~\, ij = \nj]. Here \x~\ = min{n G Z : x < n}. Since Rn = Sn{g) — M^, 



m 



one can write 



n- ^^"^ max \RA < n- max \Ru 



i<nj 



— 1/2 I I 

-\-n- max max Mj — M^^, 

■' 0<k<mj Mj<i<{k+l)ij ' ^ ' 

+ ^7^^^ max max \Si{g) - SMAg)\ ■ (3.3) 

Recalling that E(\Rn\^) = Cin^^) with a < 1/2, one can apply Corollary 3 of 
to get that for any 5 > 

P( max \Ra] > 6^) = (y{lfm^Jn,) = Q{^j~r{i~^'<'-~i^~^)P))^ 
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for any /3 > 1. Choosing /3 close enough to 1 and r large enough, the above becomes 
summable. Borel-Cantelli's Lemma implies then that the first term on the right- 
hand-side of fj3.3|) converges to 0, P-a.s. 

The second martingale term on the right-hand side of ()3.3|) tends to in P^- 
probability for vr-a.e. x, by the functional central limit theorem for L^-martingales; 
see Theorem 3 of , for example. So it all boils down to showing that the last term 
in ()3.3p goes to P-a.s. 

Remark 4. Note that we have so far used the fact that g G L'^{t^)- It is only to 
control the third term in ()3.3|) that we need a higher moment. 

Define, for 5 > 0, 

B\ = { max max \Si{g) - SkeAg)\ > Sy/n]}. 

Since g G L^{tt), one can write: 

P{B'.) < P{max\g{Xi)\ > S^/ij) 

■' i<nj 

< n,n{ \g\ > = a^r^iP/'-^^^P)). 

By choosing 7 small enough and r large enough, one can make sure that P{B'j) is 
summable. By Borel-Cantelli's Lemma, the third term in ()3.3|) converges to 0, P-a.s. 
Finally, note that if nj_i < n < rij, then 

max — — < max , (3.4) 



and so (13.211 follows. □ 



Remark 5. In the above proof we only needed the martingale term in ()3.3p to 
converge in P^.-probability. The L'^-bounds of Theorem ^ imply that it actually goes 
to P-a.s., making ()3.2j) also true P-a.s. All this is of course under the assumptions 
p> 2 and ()2.H) with a < 1/2. In [3| it is shown that the same almost-sure convergence 
happens even when p = 2. 

We also have a similar result for Sn{g)'- 

Theorem 3. Assume there are p > 2 and a < 1/2 for which g G Lp{ti) and condition 
()2.H) is satisfied. Then n~^/^maxfe<„ \Ex{Sk{g))\ converges to as n goes to infinity 
for 7T-almost every x. Consequently, for n-almost every x, 

lim n~^^^ max \Sk{g) — Sk{g) \ = P^-almost surely, 

n^oo k<n 

and, therefore, 

lim A($33, = for vr-a.e. x. 

n— >oo 

The diffusion matrix D is as defined in Theorem^ 
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Before we start the proof, we need to reprove a maximal inequality of this time 
for a Markov transition operator rather than a shift. For a probability transition 
kernel Q and a function g in its domain, define T„((7, Q) = X]fc=o then have 

the following: 

Proposition 1. Let Q be a probability transition kernel with invariant measure tt. 
Let g G L'^ijr) be such that 

|2 



Tn{g,Q)Yd'K<C{g,Q)n, 
for some C{g, Q) < oo and all n > 1. Then we have 



TT ( max \Tj{g, g)| > A J < 
\ j<n J 

for all n > 1, k > 0, and A > 0. 



A2 



Proof. We will proceed by induction on k. For k = the lemma follows from Cheby- 
shev's and Jensen's inequalities, as well as the invariance of tt under Q. Let us assume 
that the lemma has been proved for some A; > 0. We will prove it for k + 1. To this 
end, choose n>l and A > 0. Let m = [v^l- Then [n/m] < m, and 



TT ( max \Tj{g, Q)\ > x] < vr ( max \Tim{g, Q)\ > A/2 ) 

V / \i<n/m J 



+ m max vr ( max \Tj+im{g, Q) - Tim{g, Q)\ > A/2 

i<n/m \j<m 

< 71 (max\Ti{Tmig,Q),Qn\ > A/2 



+ mmax7r max |Tj(Q™^, g)| > A/2 

^ 4-2^''C{T Ug,Q),Q"'W+'" 



A2 

4 ■ 26'=C(g*™^, Q)m2+2"' 
+ max ■ 



i<m. 



But one has 



j \T4TU9,Q),Qn\''dn= j \T^n{g,Q)?d7i<C{g,Q)mn 
and, therefore, C (T„^{g , Q) , Q"^) < C{g,Q)m. Similarly, 

\Tr,iQ'^g,Q)\'dn = J \Q'"'Tr,ig,Q)\'d7T < j Q'^\T^{g,Q)\^d7: 

\T^{g.Q)\^dn<C{g,Q)n. 
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Thus, C{Q^"^g,Q) < C{g,Q). Above, we have used Jensen's inequahty to bring Q 
outside the square and then the fact that vr is invariant under Q. Now, we have 

TT max I T„ (51, (5)1 > A < 



J<n J 

Since m < 2a/?2, it follows that 



vr max|T„(5(,(5)| > A < 



7<n 7 - A2 

which is the claim of the lemma, for k + \. □ 

The following is then immediate: 

Corollary 1. For any (3 > 1 there is a constant T , depending only on (3, for which 



TX 



(max|r,((?,g)| > AJ < 



13 



for all X > and n > 1. 

We can now prove the theorem. 

Proof of Theorem^ Observe that Ex{Sn{g)) = Tn{g,Q). Now, recall that nj = j^, 
for an integer r large enough. Also, for < 7 < 1 we have mj = \nj~^'\ , = InJ] . 
Then, 

n^'^^^ m8ix\Er,{Si{g))\ < n^^^^ m8ix\Er,{SkeAg))\ (3.5) 

i<nj ■' k<mi 



-1/2 

For the first term, we can use the above corollary to write 



+ n- max max \E^{Si{g)) - E^{SMAg))\. (3.6) 

k<mj Hj<j<(fc+l)£j 



vr ( n. ^^'^ max \E^{SMAg))\ > s] = vr ( n/^^ max \TkeAg,Q)\ > s ] 

\ •' k<m.j / Y k<m.j / 



< 



7r max|Tfc(T,^,((?,g),g^OI > 

\k<m,j 

TC{TeA9,Q),Q'^)m^ 



smce 



j |r„(r,^.((7,g),g^oi't^vr = ^ |T„,^.((?,g)pt/vr < c(n£,)2" < 
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by ()2.H1 . If one chooses (3 small enough and r large enough, then the term on line 
(|3.5|) goes to 0, TT-a.s., by Borel-Cantelli's Lemma. For the term on line (j3.6|) we have 



TT 



n- max max \E^{Si{g)) - E^{SkiAg))\> e 

k<mj Mj<i<(k+l)lj ■' 

< 7T ( max \Q^g\ > eJrTjjlj ) 

< n^maxTT (IQ'fi'l > ^^/nj/dj) 



i<nj 



< (y^j-r{p/2-l-yp)^^ 



IQ'gl^dn < / Q\\g\'')d7i = / Igl^dn < oo 



since 



Using Borel-Cantelli's Lemma, we get that the term on line (j3.6|) also converges to 
0, vr-a.s., if one chooses 7 small enough and r large enough. 

Therefore, we have shown that Uj ^^'^ maxj<„,^ \Ex{Si{g)) \ converges to 0, vr-a.s. The 
claim of the theorem follows then as in (j3.4p . by considering rij < n < n^+i. □ 
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